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6.1 Understanding 
Test 


Automation 


Section Overview 


In this section, we will review the following: 


- Benefits 
- Candidates for automation 
- Automation process 


Guiding Questions 


What is the difference between an automated test and 
a Manual test? 


What are the benefits of automated testing? 


What are the names of two types of tests that can be 
automated? 


Automated Tests 


An automated test can be thought of as a set of steps 
that a computer may run programmatically to test the 
functionality of the software. 


Automated testing complements, but does not replace, 
manual testing. 


Automated tests must be created by a developer, 
while manual testing can be conducted by someone 
without programming skills. 


Benefits of Automated Testing 
Automated tests help maintain stability and help find 
regressions that might occur because of code changes. 
Automated tests can be run unattended. 


Automated tests are software applications and can be 
designed and composed of other reusable code. This makes 
automated tests flexible and maintainable. 


Automation can be run on multiple configurations by using 
Microsoft Test Manager. 


Code coverage metrics can be gathered when automated 
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Potential Drawbacks of Automated 
Testing 


When code is changed or refactored, there may be 
cascading effects that will require a corresponding 
effort to change the affected automated tests. 


There may be “psychological impact” on your team if 
code changes cause many tests to fail. 


There may be a false sense of security when all tests 
pass if the test cases are not testing for the correct 
conditions. 


Common Automated Tests 


Some types of tests are almost always automated: 


Unit Tests; |n test-driven development, unit tests are 
created first; then developers write code to pass those 
unit tests. 


Load Tests; Generating a heavy load would be 
difficult for manual testers, but automated tests can 
impose a great workload with little manpower. 


Continuous Integration Tests: You can use 
continuous integration with Microsoft® Visual Studio® 
Application Lifecycle Management (ALM) to help 
ensııra That whenever code ic daveloned and checked 


Other Potential Candidates for 


eutomation Testing: Testing on multiple installed 


environments can be a very laborious task. Microsoft 
Test Manager provides capabilities of running test 
suites on different configurations by using virtual 
machines or physical machines. 


User Interface Tests; Visual Studio ALM has 
Capabilities of creating automated tests directly for the 
user interface. 


Installation Tests: You can use the lab capabilities of 
Microsoft Test Manager to set up a group of 
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Barriers to Automation 


The benefits of automation must be weighed against 
the costs. A few considerations when determining the 
amount of automation include: 

" Creating automation requires a subset of the test team to 
learn how to write code—or requires hiring developers just 
to be on the test team. 

" Code that is changing frequently is a moving target and 
will have cascading effects into test automation code 
because it will also need to be changed. 

" Code that is tightly bound to the user interface is difficult 
to test because it may require the user to interact with the 
user interface controls. 


Automation Process 


You might begin your testing cycle by creating a 
manual test case that you decide later is a good test to 
automate. You want to be able to continue to run that 
test as part of a test plan. 

1. Create an automated test and associate it with a test case 
using Visual Studio. 

2. After you create your automated test, you must check in 
your test project that contains the automated test and 
make sure that the test project is part of your build 
definition. 

3. Then you associate the automated test with a test case 
that has been added to a test plan that uses this build. 


Section Questions 
v What is the difference between an automated test 
and a manual test? 


y What are the benefits of automated testing? 


v What are the names of two types of tests that can be 
automated? 
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6.2 Test Automation 
Strategies 


Section Overview 


In this section, we will review the following: 


- Logging 
- Automation priority 


Guiding Questions 


What feature enables automatic logging in Visual 
Studio? 


Logging Data with IntelliTrace 


IntelliTrace keeps a log of the key events as your 
tests and code execute in debug mode, and also logs 
variable values at those events. 


You can step back through the history of execution 
before the test failed, inspecting the values that were 
logged. This is particularly helpful when working on 
bugs that are difficult to replicated. 


To enable IntelliTrace, on the Visual Studio Debug 
menu, choose Options and Settings, IntelliTrace. You 
can also vary the settings to record more or less data. 


Prioritizing Automation Efforts 


Creating automated unit tests is the first priority for any test-driven 
development team—developers should be creating test cases 
before they build any methods or classes. 


Likewise, integration tests become the top priority when units are 
combined into components. 


When deciding which manual tests should be automated, consider 
the following: 


Y How often will the test be run? Give priority to frequently used 
tests. 

Y How often with is the code likely to change? Frequently 
changed code will require the tests to be maintained as well, 
possibly minimizing the benefits of automation. 

Y How much time can be saved bv running the test 


Section Questions 


v What feature enables automatic logging in Visual 
Studio? 
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6.3 Writing 
Automation 


Tests 


Section Overview 


In this section, we will review the following: 


- Logic 

- Error Handling 

- Commenting 

- Virtual Machines 


Guiding Questions 


How is a virtual machine different from a physical 
machine? 


What is the name of a condition test used to check 
logic in debug builds? 


What happens to these condition tests in a release 
build? 


Testing Logic with Assertions 


An Assertion is a statement that tests a condition during program 
execution. If the assertion is true, no action occurs; if the assertion 
IS false, the assertion fails. 


In a debug build, a failed assertion causes the program to enter 
break mode, making it easier to examine the nature of the problem. 


In a release build, assertions are ignored. 


When you use an assertion, make sure that any code inside the 
assertion block does not change the results of the program if the 
assertion is removed. In other words, it should be only non-essential 
code. 


Y Otherwise, you might accidentally introduce a bug that only shows 
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Errors in Testing 


Because automated tests consist of code, it is possible for the 
tests themselves to have errors. 


If a test fails to run, you can investigate the failure by checking 
the test environment; this includes the way the test is set up 
and the settings in the active test settings. 


In some cases, such as those related to deployment, failures 
are independent of test type. In other cases, the test type 
determines how and what to investigate. 


Errors that involve tests are reported to you at either of two 
levels: 


" Test-level errors. An example of a test-level error is a test timeout 
error, which occurs if the test's timeout limit is reached. 


" Run-level errors. An example of a run-level error is a run timeout 


Test Comments 


While you run a test using Test Runner, you have the 
option to add comments to the test steps. 


The comments can be used to relay specific details 
that you want to communicate to other team members 
regarding unexpected behavior, or suggestions about 
what to look for while you test an application. 


These comments can be useful as supporting material 
related to bugs, for isolating code defects, or for 
refining user interface designs. 


Virtual Machınes 


In general, a computer system uses a single operating system. Since 
developers may want to test an application across many different 
operating systems for compatibility, this would necessitate having 
multiple computer system with various configurations, a potentially 
impractical approach. 


However, you can set up a software environment that emulates an entire 
separate computer system by handling instructions in the same way a 
native processor (or “physical machine”) would handle instructions. This 
is called a virtual machine (VM). 


By setting up a VM, you can test a different configuration without 
purchasing an additional computer. 


Modern computer systems may even be capable of running multiple VM’s 
simultaneously. 


Section Questions 


Y How is a virtual machine different from a physical 
machine? 


v What is the name of a condition test used to check 
logic in debug builds? 


v What happens to these condition tests in a release 
build? 
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6.4 Managing Test 
Scripts 


Section Overview 


In this section, we will review the following: 


- Smoke test 
- Build verification test 
- Lab Management 


Guiding Questions 


What are two types of lab environments? 
What does BVT stand for? 


What type of machines can a SCVMM environment 
manage? 


Smoke Testing 


In software, the term smoke testing describes the 
process of validating code changes before the changes 
are checked into the product's source tree. 


After code reviews, smoke testing is the most cost 
effective method for identifying and fixing defects in 
software. 


Smoke tests are designed to confirm that changes in 
the code function as expected and do not destabilize 
an entire build. 


In chort emoke tacte fociis on the featııres of the 


Build Verification Tests (BVT) 


A smoke test is also known as a Build Verification Test 
(BVT) 


In this case, it gets its name because it is typically 
executed after a daily (or other “short term”) build. 


The following steps are necessary to execute BVIs in 
Microsoft® Visual Studio®: 

v Check in the newly created/modified test(s) 

v Create a build definition 

Y Set up any necessary test agents 

v Use Team Explorer to queue up and start the test(s) 


Lab Environments 


A thorough testing strategy includes tests conducted 
across multiple systems, perhaps with different 
hardware and operating systems. 


Rather than manually re-configuring each machine for 
each set of tests, Visual Studio Lab Management 
handles configuration, deployment, and testing. 


A lab environment is a collection of computers that 
are managed as a single unit, and on which you deploy 
the system under test along with test software. 


Advantages of a Lab Environment 


The results of tests can be shown on charts that relate 
them to system requirements. 


Lab Manager automatically installs test agents on each 
machine, enabling test data to be collected. 


You can view the consoles of the machines through a 
single viewer, switching easily from one machine to 
the other. 


Lab environments manage the allocation of machines 
to tests for reasons that include preventing two team 


Two Types of Lab Environments 


Standard lab environment, which can contain a mix 
of virtual and physical machines. 


SCVMM environment, which can only contain virtual 
machines that are controlled by System Center Virtual 
Machine Manager (SCVMM). 


Features Avallable in SCVMM 
ENIRA Shots: Environment snapshots contain the 


state of a lab environment, so you can quickly restore a clean 
environment, or save the state of an environment that has 
been modified. 


Network isolation: Network isolation allows you to 
simultaneously run multiple identical copies of an SCVMM 
environment without computer name conflicts. 


Virtual machine templates: A virtual machine template is a 
virtual machine that has had its name and other identifiers 
removed. This allows you deploy multiple copies of a virtual 
machine in the same environment, or multiple environments, 
and then run the virtual machines simultaneously. 
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Supported Operating Systems for VM 


* Windows® XP SP3 and later versions 
* Windows Vista® 

* Windows Server® 2003 

* Windows Server® 2008 

* Windows Server® 2008 R2 

* Windows Server® 2012 

* Windows 7 

* Windows 8 or later version 


Section Questions 


v What are two types of lab environments? 
v What does BVT stand for? 


v What type of machines can a SCVMM environment 
manage? 


Additional Resources for this Module 


Walkthrough: Creating and Running a Load _http://msdn.microsoft.com/en- us/library/dd537628.aspx 


Test Containing Web Performance Tests 
Walkthrough: Creating a Simple Web App 


Walkthrough: Recording and Running a 
Web Performance Test 


Using Code Coverage to Determine How 
Much Code is being Tested 


Unit Testing: Testing the Inside 


Debug Your App by Recording Code 
Execution with IntelliTrace 


Walkthrough: Creating and Running a 
Generic Test 


Assertions in Managed Code 


Lab Environments 
Troubleshooting Test Execution 


Walkthrough: Managing Tests Using Lists 
and Properties 
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http://msdn.microsoft.com/en-us/library/ms182551.aspx 


http://msdn.microsoft.com/en- us/library/dd537628.aspx 


http://msdn.microsoft.com/library/jj 15934 0.aspx 


http://msdn.microsoft.com/en- 
uS/library/vstudio/dd264915.aspx 


http://msdn.microsoft.com/en-us/library/ms182626.aspx 


http://msdn.microsoft.com/en- 
uS/library/ttcc4x86(v=vs.110).aspx 


http://msdn.microsoft.com/en- us/library/jj159341.aspx 
http://msdn.microsoft.com/en- us/library/ms182478.aspx 


(http://msdn.microsoft.com/en-US/library/ 
ms182466(v=vs.100).aspx 
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